Efficient Nearest Neighbor Indexing Based on a Collection of Space Filling Curves

A database is populated with a set of points represented by n-tuples of real numbers. A query consists of a point q (not necessarily in the database) and an integer k, asking for the k "nearest" database points to the query point. The exact output for the query consists of the k nearest points, but if the database is large and a quick response is required, a good approximate output is sought. All currently known methods require at query time calculation of distances fiom the query point to many database points. The computational effort is dominated by the number of such distance calculations since points have to be fetched fiom random locations in the database and the high dimension implies that current database indexes cannot significantly restrict the number of points that have to be fetched. In many cases, a complete linear scan of the database beats the currently known methods. The number of such distance calculations performed by current methods grows with the number of points in the database. The method described in this report has shown (in experiments on databases with tens of thousands of points with hundreds of dimensions, and asking for about 100 nearest neighbors) to provide very good approximate output sets while limiting the number of distance calculations to a few hundreds. Theoretical analysis predicts that the number of required distance calculations depends on the dimension and not on the number of points in the database. Therefore, when more points are added to a database the dominant factor in the query effort does not change. Efficient Nearest Neighbor Indexing Based on a Collection of Space Filling Curves Nimrod Megiddo and Uri Shaft IBM Almaden Research Center October 20, 1997